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John  R.  Anderson 


When  we  began  the  contract  we  had  developed  an  effective  intelligent  tutor  for  the  Instruction  of 
programming  skills  in  LISP.  The  state  of  this  tutor  and  our  general  thinking  about  tutoring  systems  is 
described  in  the  following  report,  which  despite  its  1990  publication  date,  reports  the  state  of  our  tutoring 
work  as  of  the  end  of  1 986: 

Anderson,  J.  R.,  Boyle,  C.  F.,  Corbett,  A.,  Lewis,  M.  W.  (1990)  Cognitive  modelling  and  intelligent  tutoring. 
Artificial  Intelligence,  42,  7-49. 

Our  goal  in  the  contract  was  to  understand  the  success  that  we  had  created.  There  were  two  basic 
directions  for  exploring  this.  One  was  to  examine  In  detail  the  behavior  of  students  with  the  LISP  tutor 
and  see  what  light  that  behavior  shed  on  basic  issues  of  skill  acquisition.  The  second  direction  was  to 
explore  variations  on  the  LISP  tutor  to  see  what  would  happen  as  we  changed  various  aspects  of  the 
tutor. 

Detailed  Studies  of  the  Course  of  Skill  Acquisition 

One  major  line  of  analysis  in  the  first  direction  was  to  examine  the  protocols  that  the  LISP  tutor  gave  us 
recording  the  student  interactions.  These  protocols  were  classified  according  to  the  production  rules 
which  were  involved  in  the  solution  of  the  problem  by  the  ideal  model.  This  allowed  us  to  track  the 
acquisition  and  performance  of  individual  production  rules.  The  three  reports  describing  this  research,  in 
order  of  their  creation  dates  (not  publication  dates)  are: 

Anderson,  J.R.  (1990).  Analysis  of  student  performance  with  the  LISP  tutor.  In  N.  Fredericksen, 
R.  Glaser,  A.  Lesgold,  &  M.  Shaffo  (Eds.),  Diagnostic  Monitoring  of  Skill  and  Knowledge  Acquisition. 
Hillsdale,  NJ:  Erlbaum,  27-50. 

Conrad,  F.  G.  &  Anderson,  J.  R.  (1988).  The  process  of  learning  LISP.  Cognitive  Science  Meetings, 
454-460. 

Anderson,  J.  R.,  Conrad,  F.  G.,  &  Corbett,  A.  T.  (1989).  Skill  acquisition  and  the  LISP  Tutor.  Cognitive 
Science,  13,  467-506. 

We  found  that  individual  production  rules  followed  the  learning  history  attributed  to  them  by  the  ACT* 
theory.  In  particular,  they  showed  rapid  improvement  upon  first  practice  opportunity  (reflecting  knowledge 
compilation)  followed  by  more  gradual  improvement  (reflecting  production  rule  strengthening).  The 
gradual  improvement  process  was  described  by  a  power  function.  We  also  found  evidence  for  an 
influence  of  problem  load  on  production  rule  performance.  This  problem  load  effect  diminished 
throughout  the  experiment  as  the  problem  became  practiced.  We  also  examined  the  nature  of  individual 
differences  in  performance  with  the  LISP  tutor  and  only  found  evidence  for  two  general  ability  factors 
which  we  called  an  acquisition  factor  and  a  retention  factor. 

The  analysis  of  production  rule  practice  applied  both  to  the  dependent  measures  of  time  and  errors.  One 
source  of  errors  reflects  slips  in  Are  performance  of  rules.  However,  another  source  is  errors  in  the 
analogy  process  that  underlies  knowledge  compilation.  We  were  able  to  show  that  high  frequency  errors 
tend  to  have  their  origins  in  analogy  In  the  followlnp  report: 

Anderson,  J.  R.  (1989).  The  analogical  origins  of  errors  in  problem  solving.  In  D.  Klahr  &  K.  Kotovsky 
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(Eds.),  21st  Carnegie  Symposium  on  Cognition,  343-371. 

Studies  of  tutor  characteristics 

The  other  direction  of  research  concerned  studies  of  the  characteristics  of  the  LISP  tutor  itself.  The  two 
major  dimensions  we  manipulated  were  the  timing  and  content  of  feedback  to  the  students.  With  respect 
to  timing  of  feedback  we  have  created  four  different  tutoring  disciplines.  One  is  the  original  immediate 
feedback  tutor  where  students  are  corrected  as  soon  as  they  make  a  mistake.  The  second  is  a  delayed- 
feedback  condition  where  they  only  receive  feedback  when  they  complete  a  problem.  The  third  is  a 
demand  feedback  condition  where  students  can  ask  for  feedback  at  any  point  they  like  in  the  problem 
solution.  The  fourth  is  the  flag-tutor  where  the  tutor  signals  when  the  student  makes  an  error  but  the 
student  can  choose  when  and  if  to  get  feedback  on  the  error.  These  four  tutors  are  listed  above  in  the 
order  of  their  creation.  Reports  of  studies  on  these  tutors  are  contained  in 

Corbett,  A.  T.,  Anderson,  J.  R.,  &  Patterson,  E.  J.  (1988).  Problem  compilation  and  tutoring  flexibility  in 
the  LISP  Tutor.  Intelligent  Tutoring  Systems,  423-429. 

Corbett,  A.  T.,  Anderson,  J.  R.,  &  Patterson,  E.  G.t  (in  press).  Student  modelling  and  tutoring  flexibility  in 
the  LISP  Intelligent  Tutoring  System.  In  C.  Frasson  and  G.  Gauthier  (Eds.)  Intelligent  tutoring  systems:  At 
the  crossroads  of  artificial  intelligence  and  education.  Norwood,  NJ:  Ablex. 

Corbett,  A.  T.  &  Anderson,  J.  R.  (in  press).  The  LISP  intelligent  tutoring  system:  Research  in  skill 
acquisition.  In  J.  Larkin,  R.  Chabay,  C.  Scheftic  (Eds.),  Computer  Assisted  Instruction  and  Intelligent 
Tutoring  Systems:  Establishing  Communication  and  Collaboration.  Hillsdale,  NJ:  Eribaum. 

Corbett,  A.  T.,  &  Anderson,  J.  R„  (1989).  Feedback  timing  and  student  control  in  the  LISP  Intelligent 
Tutoring  System,  Artificial  Intelligence  and  Education,  64-72. 

Corbett,  A.  T.  &  Anderson,  J.  R.  (1990)  The  effect  of  feedback  control  on  learning  to  program  with  the 
Lisp  Tutor.  Proceedings  of  the  Twelfth  Annual  Conference  of  the  Cognitive  Science  Society,  Cambridge, 
MA. 

Students  tend  to  turn  the  demand  feedback  condition  into  a  delay  feedback  condition,  waiting  until  the 
end  of  the  problem  to  get  feedback.  On  the  other  hand,  they  tend  to  turn  the  flag  tutor  into  the  immediate 
feedback  tutor  correcting  90%  of  their  errors  almost  immediately.  However,  most  of  the  time  they  correct 
their  flagged  errors  without  requesting  feedback  from  from  the  the  tutor.  Students  take  longer  in  the 
delayed  feedback  conditions  than  the  Immediate  feedback  condition-as  much  as  four  times  longer. 
Usually,  we  do  not  find  any  achievement  differences  on  tests  of  final  performance.  When  there  are 
substantial  time  pressures  on  students  which  cause  them  to  cut  comers  and  not  study  solutions  in  the 
delay  conditions,  we  do  find  these  conditions  result  in  poorer  final  achievement.  However,  it  seems  that  if 
students  have  enough  time  to  solve  all  the  problems  and  understand  them  their  final  achievement  levels 
do  not  depend  on  the  tutoring  mode.  This  Is  Interesting  because  students  can  go  through  rather  different 
trajectories  in  solving  the  problems  in  the  different  modes.  It  seems  that  learning  is  a  function  of  the 
product  of  problem-solving  and  not  the  process. 

The  work  on  content  of  feedback  is  reported  in 

Anderson,  J.  R„  Conrad,  F.  G„  &  Corbett,  A.  T.  (1989).  Skill  acquisition  and  the  LISP  Tutor.  Cognitive 
Science,  13,  467-506. 
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Corbett,  A.  T.  &  Anderson,  J.  R.  (In  press).  The  LISP  intelligent  tutoring  system:  Research  in  skill 
acquisition.  In  J.  Larkin,  R.  Chabay,  C.  Scheftic  (Eds  ),  Computer  Assisted  Instruction  and  Intelligent 
Tutoring  Systems:  Establishing  Communication  and  Collaboration.  Hillsdale,  NJ:  Erlbaum. 

We  found  that  providing  students  with  explanatory  error  messages  helped  them  immediately  correct  their 
errors  but  had  no  long-term  benefits  in  terms  of  final  achievement  tests.  We  think  that  perhaps  part  of  the 
reason  is  that  such  explanations  deny  students  the  opportunity  to  explain  their  own  errors.  In  the  flag 
tutor  subjects  are  able  to  correct  up  to  90%  of  their  errors  without  requesting  an  explanation  from  the 
tutor. 

Conclusions 

We  have  emerged  from  this  contract  with  a  characterization  of  skill  acquisition  which  is  very  much  in 
keeping  with  the  ACT.  This  outcome  plus  the  results  of  our  tutor  comparisons,  leads  us  to  believe  that 
something  on  the  order  of  the  flag  tutor  may  reflect  the  optima)  tutoring  style.  It  provides  the  efficiency  of 
learning  associated  with  the  original  immediate  feedback  tutor  while  providing  the  advantages  of 
discovery  and  self-explanation  in  those  situations  where  subjects  are  capable  of  such  activities. 
Philosophically,  it  reflects  a  desirable  approach  to  instruction  in  which  the  tutor  does  not  hide  from  the 
student  its  assessment  while  at  the  same  time  not  forcing  instruction  or  a  particular  solution  style  on  the 
student.  While  there  are  all  these  reasons  to  be  positive  about  the  flag  tutor  we  have  not  yet  done  a 
study  that  shows  a  statistically  significant  advantage  of  the  flag  tutor  in  terms  of  learning  time,  final 
achievement,  or  student  ratings  of  the  final  outcome.  All  comparisons  to  date  of  the  flag  tutor  with  the 
Immediate  feedback  tutor  produce  effects  below  the  level  of  statistical  significance.  The  one  dear 
advantage  is  that  it  receives  much  more  positive  evaluations  from  other  researchers. 
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